Indirect and Direct Input Devices
Typically, a computer implemented graphical application is controlled with a pointing input device. Indirect pointing devices include a mouse, a trackball and a touch pad, and direct pointing devices include touch sensitive display surfaces. The interaction with graphical objects is typically represented by a graphical pointer displayed by the application on an output device. Movement of the input device results in corresponding movement of the pointer on the output device.
With an indirect input device, such as a mouse, touch pad, or trackball, the input device and the pointer are physically separate. Moreover, the input device typically moves on a horizontal plane of a desktop or touch pad, while the pointer moves on a vertical plane of the display device. When using a direct pointing input device such as a touch-sensitive surface, graphical objects are manipulated directly by touching them on the display surface. Direct input modalities are a viable alternative to indirect input, such as using a mouse.
Absolute and Relative Mapping
The mapping between a position of the pointing device and a location of the pointer can be either absolute or relative. Most direct pointing input devices use an absolute mapping. That is, the pointer is positioned directly under a finger or a stylus used for the pointing. While this is the most obvious, and arguably the most natural mapping, there are drawbacks. Hands, arms, fingers, and stylus can occlude portions of the display. This can be a nuisance in both single and multi-user systems.
For many rear-projection output devices, accurate pointing and selection are hindered by parallax error. For front-projection, the finger or hand casts a shadow over the object of attention.
On a small display all areas of the user interface are easily within reach, on a large display this may not be the case. Using direct input devices loses its desirability the more users must stretch their arms, twist their body, or physically walk to distant parts of a display. In extreme cases, it may become impossible to reach the entire extent of very large displays. These difficulties only increase when working in a large, multi-display environment in which distant graphical objects may not only be hard to reach, but also may require the user to interact across bevels or even gaps between displays. For example, directly moving a pointer on a display surface that is several square meters large is awkward, and perhaps physically impossible.
Indirect pointing overcomes many of these limitations, albeit perhaps at the cost of naturalness. The main benefit in terms of large, wall-sized displays is that distant targets can be manipulated without walking, as small movements of the indirect input device can be mapped to large movements of the pointer. The opposite is also true. Relative mapping with indirect input devices allows for more control over pointing because control-display (CD) gain ratios can be less than 1:1 for slow movements and more accurate pointing.
Occlusion and parallax error are also less of a problem with indirect pointing. In addition, dragging pointers between different displays is easily supported, and all areas of a large display are easily selected. In multi-user setting, users can reposition themselves such that they do not block the view of collaborators.
Regarding the performance of indirect and direct pointing, the prior art describes different conclusions. Sears et al. compared indirect mouse input to direct touch screen input. Their experiment used a 27.6 by 19.5 cm display with a mouse CD gain close to 1. For targets that are 16 pixels in width and greater, they found direct selection using a touch-sensitive screen was faster than indirect selection with a mouse. Furthermore, for targets 32 pixels in width of greater, direct touch selection resulted in about 66% fewer errors than with an indirect mouse. Yet, even with the apparent superior performance of direct touch input modality, participants still preferred mouse input, Sears, A. and Shneiderman, B., “High Precision Touchscreens: Design Strategies and Comparisons with a Mouse,” International Journal of Man-Machine Studies, 34(4). 593-613, 1991.
Meyer et al. also compared direct input devices with indirect input devices. They found that when used in an indirect manner, with physically separated control and display areas, the indirect device performed better than the direct device. In fact, they found all direct input devices to be slower than the indirect devices, Myers, B., Bhatnagar, R., Nichols, J., Peck, C. H., Kong, D., Miller, R. and Long, C., “Interacting at a distance: measuring the performance of laser pointers and other devices,” in Proc. of ACM CHI Conference on Human Factors in Computing Systems, pp. 33-40, 2002.
On the other hand, Graham et al. compared selection performance using direct physical and indirect virtual touching. In the physical mode, users selected targets with their hand directly on a physical surface, but in the virtual mode, the user's hand was hidden and rendered as a “virtual finger” on a display. There was no performance difference between techniques for the initial movement phase, but virtual touching was slower in the second movement phase as the hand decelerated to select small 3 to 12 mm targets. This suggests that direct input can outperform indirect input in some situations, Graham, E. and MacKenzie, C. Physical versus virtual pointing. in Proc. of the CHI '96 Conf. on Human Factors in Computing Systems, pp. 292-299, 1996.
Those results are in contrast to the results described by Accot et al. They found that for steering tasks users were about twice as fast with an 8″×6″ indirect touch tablet in absolute mode, than with a smaller indirect touchpad in relative mode, Accot, J. and Zhai, S. Performance evaluation of input devices in trajectory-based tasks: an application of the steering law. in Proc. of ACM SIGCHI Conference on Human Factors in Computing Systems, pp. 466-472, 1999.
One absolute, indirect input device is described by, J. K., Mandryk, R. L., and Inkpen, K. M., “TractorBeam: seamless integration of local and remote pointing for tabletop displays,” in Proceedings of the 2005 conference on Graphics interface, Human-Computer Communications Society, pp. 33-40, 2005. That system uses a handheld Polhemus Fastrak tethered pointing device for six degrees of freedom motion tracking. The displayed pointer, in this case a virtual laser dot, is always displayed on a tabletop exactly where the handheld pointer is aimed. However, accurate selection of distant targets with a handheld devices is well known to be difficult and error prone. It is impossible to hold a hand steady for an extended length of time. Jitter is inevitable. This makes the usability of handheld direct input devices questionable for large displays, Myers, B., Bhatnagar, R., Nichols, J., Peck, C. H., Kong, D., Miller, R. and Long, C., “Interacting at a distance: measuring the performance of laser pointers and other devices,” in Proc. of ACM CHI Conference on Human Factors in Computing Systems, pp. 33-40, 2003.
There is also an issue of reach with large displays or across multiple displays. Many prior art solutions repurpose large areas of the workspace or display graphical feedback over a wide area of the display. While that approach is fine for an individual user on a wall-sized display, the heavy use of graphics may be inappropriate for multi-user workspaces in which other users may become distracted by these techniques.
Baudisch et al. described Drag-and-Pop as a means of moving a selected object to a distant target. When that technique is invoked, proxies for distant targets are drawn near the user where the proxies are easily within reach. Reference lines connect these proxies to their true targets. However, there is a good chance that these reference lines might cut through another's workspace in a multi-user setting. Additionally, Baucisch's technique does not allow for the selection of the area around target objects, only the selection of the objects themselves. Baudisch, P., Cutrell, E., Robbins, D., Czerwinski, M., Tandler, P., Bederson, B. and Zierlinger, A., “Drag-and-Pop and Drag-and-Pick: Techniques for Accessing Remote Screen Content on Touch and Pen operated Systems,” in Proceedings of the ninth IFIP TC13 international conference on Human-Computer Interaction, pp. 57-64, 2003.
Similarly, Bezerianos et al. described the Vacuum technique for selecting distant objects. The Vacuum displays a large adjustable area of effect that can easily cover much of the display as distant targets are drawn close to the user. Like Drag-and-Pop, the disruption of other users working in the same space may reduce the benefits of addressing the reachability problem of large displays, Bezerianos, A. and Balakrishnan, R., “The vacuum: facilitating the manipulation of distant objects,” in Proceedings of the SIGCHI conference on Human factors in computing systems, ACM Press, pp. 361-370, 2005, incorporated herein by reference.
FIG. 1A shows a three state model for a direct input and absolute only mapping system, Buxton, W., “A three-state model of graphical input,” in Proc. of the IFIP TC13 Third International Conference on Human-Computer Interaction, pp. 449-456, 1990. That system uses a stylus with a tip switch, and a horizontal display surface. When the stylus is not in contact with the surface the system is in State 0 101. The tip switch is open, and the stylus is considered to be out of range. Movement of the stylus has no effect. When the stylus is on the surface, a displayed pointer absolutely tracks the movement of the stylus and the system is in State 1 102. When the stylus is pressed hard on the surface, the tip switch is closed, and a displayed graphical object coincident with the pointer is absolutely dragged in State 2 103. Removing pressure on the stylus, while keeping the stylus in contact with the display surface opens the tip switch and ‘releases’ the graphic object, and the system reverts to State 1. Lifting the stylus cause the system to enter State 0. This emulates the actions of an indirect input device, such a mouse, where closing the switch has the same effect as pressing the left button on the mouse while moving the mouse.
For relative mapping, one needs to support not only tracking, and dragging/selection, but also clutching. FIG. 1B shows a modified three-state model for a mouse pen with relative only mapping. The mouse pen is operated by moving the tip of the mouse pen on a surface. The mouse pen also includes a switch. When the mouse pen is in light contact with surface the switch is open, and the displayed pointer tracks the movements of the pen, but unlike the absolute stylus in FIG. 1A, the graphical pointer is not directly under the tip of the pen. This is a relative tracking state 104. Lifting the pen up and slightly away from the surface signals a clutching action, which is referred to as the clutching state 103. Putting the pen in contact with the surface returns to the tracking state 104, except that the pointer now moves relative to where the pointer was before the clutching action took place. This is similar to lifting a mouse from the work surface, and placing the mouse elsewhere. The pointer initially remains at the location where it was, and then can be moved again with the mouse at a different position. Pressing the pen onto the surface closes the switch and selects a graphical object under the graphical pointer and drags the object. This is the dragging state 105.
When tracking and dragging, the CD gain between the input device and pointer movement can vary as a function of stylus velocity. Typically, this is referred to as a pointer acceleration function.
Intuitively, absolute mapping should perform well when distances to be traversed are small, whereas a relative mapping is best when distances are large. However, the affordance of an absolute “under-the-stylus” mapping may be so strong that users could find using a relative mapping difficult or unnatural, lowering performance even at large distances. Further, using a relative mapping for a graphical object that is far away might result in the object is being harder to see and select than in an absolute mapping mode where the user is always visually close to the object.
FIG. 1C shows the results of an experiment in which user performance is compared for an object target selection task between absolute (FIG. 1A) and relative (FIG. 1B) mapping modes for direct input on a large wall-sized display. The graphs in FIG. 1C show selection times for targets at different distances for both absolute and relative mapping. The crossover in performance between absolute and relative mapping occurs at distances of about two meters. Object distances in this experiment range from about 1000 pixels to almost 4000 pixels of the display, which corresponds to physical distances between one to four meters.
As one can see from the graph, users perform better when using absolute mapping for nearby objects, and performed better when using relative mapping for distant objects. This crossover in performance indicates that when working on a single large display, which has a cross sectional width larger than two meters is over 2 m, users may benefit from being able to select an absolute or relative mapping when using their input device to work with differently distances.
In a similar experiment, participants used both absolute and relative mapping on a TabletPC with a 12.1″ diagonal display. Participants were significantly faster at and greatly preferred selecting targets with absolute mapping for objects on this small display. However, for small objects, users were more accurate when using relative mapping.
This tradeoff among speed, preference, and accuracy indicates that users may benefit from being able to switch to relative mapping when a high level of accuracy is required.
Multi-display environments, such as that described by Streitz et al. have generated a lot of interest in recent years. These environments often include a heterogeneous mix of devices with different input capabilities, Streitz, N., Geiβler, J., Holmer, T., Konomi, S., Müller-Tomfelde, C., Reischl, W., Rexroth, P., Seitz, P., and Steinmetz, R., “i-LAND: An interactive Landscape for Creativity and Innovation,” in Proc. of the ACM Conf. on Human Factors in Computing Systems, pp. 120-127, 1999. Enabling a user to perform input with absolute mapping when working on their personal laptop or handheld systems, while allowing the users to switch to relative mapping for controlling pointers on other displays is desirable.
Johanson et al. described a pointing technique, called PointRight, Johanson, B., Hutchins, G., Winograd, T., Stone, M., “PointRight: Experience with Flexible Input Redirection in Interactive Workspaces,” in Proc. of the ACM Conf. on User Interface and Software Technology, pp. 227-234, 2002. That technique allows a user to move a system pointer across the displays of multiple systems using a single indirect device, such as a mouse. In that case, a user can manipulate graphical objects displayed on a handheld as well as objects displayed on wall mounted displays. Users can also be positioned around a touch-sensitive table surrounded by large vertical displays. While nearby displays in such a multi-display setting could benefit from direct input, it would be tedious to force users to walk to distant displays in order to interact with the graphic objects. On the other hand, indirect input would allow users to work with distant displays, but would be awkward for objects on nearby displays.
Up to now, system designers must select between an absolute direct input modality or a relative indirect input modality when implementing systems. However, it is desirable to provide an input mechanism that enables fluid switching between relative and absolute mappings, while using a direct input device, thus, enabling users to benefit from the best of both input modalities.